On Practical Algorithms for Entropy Estimation and the Improved Sample Complexity of Compressed Counting

Abstract The long-standing problem of Shannon entropy estimation in data streams (assuming the strict Turnstile model) is now an easy task by using the technique proposed in this paper. Essentially speaking, in order to estimate the Shannon entropy with a guaranteed ν-additive accuracy, it suffices to estimate the αth frequency moment, where α = 1−∆, with a guaranteed ǫ-multiplicative accuracy, where ǫ = ν∆. Previous studies have shown that ∆ has to be extremely small (e.g., ∆ < 10 or even much smaller). In other words, the sample complexity for entropy estimation is O ( V ǫ2 )